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Systems Engineering Programmatic Estimation Using Technology 

Variance 

Abstract 


Unique and innovative system programmatic estimation is conducted using the variance of 
the packaged technologies. Covariance analysis is performed on the subsystems and 
components comprising the system of interest. Technological "return" and "variation" 
parameters are estimated. These parameters are combined with the model error to arrive 
at a measure of system development stability. The resulting estimates provide valuable 
information concerning the potential cost growth of the system under development. 

1.0 Introduction 

Systems engineering (SE) is a field that is generally viewed as the process of formulating 
and solving problems at a very high conceptual level. SE covers all phases of the life cycle 
of a system. One phase of particular interest is that of system development. The 
conceptual design of a system may or may not resemble the design at the end of the 
system development phase. Moreover, the cost and schedule parameters established at the 
beginning of development can vary significantly from those at system delivery. However, 
the basic technologies for the final design are usually established fairly early on in the SE 
process. What information can be learned from these technologies early on that will be 
useful for long-term prediction in the system development phase? 

The Complex Organizational Metric for Programmatic Risk Environments (COMPRE) 
tool utilizes technology return and variance in synthesizing programmatic elements into 
predictions of expected cost growth as a measure of programmatic success. However, 
COMPRE does not directly utilize technology return and variance in arriving at 
probability statements concerning programmatic cost growth. 


1.1 Study Objective 

The objective of this study is to improve the understanding of the role of technology 
return and variance parameters in predicting probability of success for system development 
programs This effort will help answer the following key questions: 

1 . Can technology return and variance parameters be used to gain insight into systems 
development measures of performance, specifically probability of cost and schedule 
growth? 

2. If so, what parameters are significant, and how can they be used? 

3. Do the results satisfy face validity? 
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1.2 Expected Significance 


The expected significance of this research and development effort includes an improved 
understanding of systems engineering design and development through the quantification 
of probability measures of performance relating to technology return and variance 
parameters 

1.3 Statement of Uniqueness 

The combination of technology return and variance parameters with neural networks to 
produce meaningful probability statements has not been previously applied to large 
complex programs of the type that NASA frequently encounters. 


2.0 Technical Results 

2.1 Derivation/Implementation of Technology Return and Variance 
Parameters 

In this section, the method for utilizing technology return and variance parameters in the 
prediction of probability of success for programs is developed. The method of application 
is implemented and verified using the available database and existing Pilot Study. 

2.1.1 Current Prediction Method 

Figure 2. 1.1-1 provides the current configuration of the COMPRE model. Results from 
the covariance analysis feed directly into the artificial neural network. This neural net 
predicts expected cost growth. The expected cost growth, in combination with the neural 
net model error, results in an "s-curve" prediction of probability of cost growth. Note, in 
particular, that there is no direct link between the covariance analysis and the "s-curve." 

The mathematics of the "s-curve" prediction show clearly this lack of a direct link. The 
equation for prediction is given by: 


P{CG < TCG ) = NORMS{ ~ Mnn ) 

<J NN 


where, 

CG = actual (unknown) programmatic cost growth (%), 
NORMS = standard normal distribution, 

P = probability, 

TCG = target cost growth (%), 

Pnn = expected cost growth (%) from neural net. 
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Onn = neural net average error (%) in predicting cost growth 



Figure 2.1. 1-1. Current COMPRE Methodology Configuration 


2.1.2 Expression of Probability Statement Using Technology Return and Variance 

A direct link must be provided between the covariance analysis of Figure 2. 1.1-1 and the 
"s-curve" probability statement of the same figure. Thus, the probability of cost growth 
would depend not only on the output and modeling error of the neural net, but also on the 
"return" and variance (more specifically, standard deviation) of the technology package 
utilized in the construction of the system. The general equation for such a linked 
relationship is given below. It was the purpose of this study to determine and evaluate the 
form of the function, f, in this relationship. An important consideration in determining the 
form of f is in the assumption of dependency among the "means" and "standard deviations" 
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involved in this mathematical expression. For instance, are the neural net and covariance 
standard deviations independent of one another? What about the means? 


P(CG < TCG ) - f( TCG, , <J NN , /j c , <7 c ) 


where, 

CG = actual (unknown) programmatic cost growth (%), 
f = function, 

P = probability, 

TCG = target cost growth (%), 

Pc = expected value from covariance analysis, 

Pnn = expected cost growth (%) from neural net, 

(Jc = standard deviation from covariance analysis, 

(Jnn = neural net average error (%) in predicting cost growth. 


In fact, we may write the desired probability statement as the standard normal distribution 
calculation: 


P(CG < TCG ) = NORMS ( — — — 

(J y 


where, 

CG = actual (unknown) programmatic cost growth (%), 

NORMS = standard normal distribution, 

P = probability, 

TCG = target cost growth (%), 

Pi = expected total cost growth (%), 

dr = total average error (%) in predicting cost growth. 

Then, the question becomes how to calculate the total mean and standard deviation as a 
function of those from the covariance analysis and neural net. If we merely were in a 
situation where the sum of the random variables (from the covariance and neural net) were 
of interest, then we would write: 

Mt=Rc+ Mm 
2 2 2 

g T - a C + a NN + Pc ,NN G C G NN 

where, again, the subscripts C and NN refer to covariance and neural net, respectively, 
and Pc,nn is the correlation between the covariance and neural net data. However, we are 
not in a situation in which addition of the random variables is appropriate, since the 
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covariance analysis is an intermediate step in calculating the neural net results. Thus, we 
are in a situation in which measurement error is the key consideration. The prediction of 
cost growth from the neural net is still the best available estimate of expected value, but 
the neural net error is only one contributor to the standard deviation of that estimated cost 
growth. Thus, we see from Juran that: 

Ml = Mnn 

G T = <7 Q + (Tjjn + Pt\NN <J C CJ NN 

Now, we must determine how to calculate the correlation, Pqnn. We may estimate this 
correlation as the correlation between the covariance expected value and the neural net 
expected value taken over the entire database of programs: 

Pc.NN ~ P(PC > MnN ) 

or 

Pc,nn = 0-628 

Then, our final form becomes: 

M t ~ Mm 

2 2 2 

(J Y — Gq + (J 0.628(j ^(7 pj jy 


'Ty^/^r 

P(CG < TCG) = NORMS ( —. — - OM 

I 2 2 

yjcrc +&m + 0 . 62 ^(j c (t nn 


2.1.3 Application to MSRR-1 Pilot Study 

For the MSRR-1 program, we have the following pertinent values: 


M-nn - 1 2.6 %, 
(Jc = 13.64 %, 
Onn = 5.0 %. 


Substituting into the equation above, we get: 


P(CG < TCG ) = NORMS ( 


TCG - 12.6 
15.93 


) 
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From Figure 2. 1.3-1, we see that the 95% confidence level for cost growth is 
approximately 39%. The 50% confidence level is, of course, equal to the expected value 
of the neural net output, or a cost growth of 12.6%. 



Figure 2.1.3-1. MSRR-1 Cost Growth Profile 


3.0 Conclusions 

1 . The neural net estimate for expected cost growth of the program of interest remains 
the best estimate for same. This estimate is unaltered by the presence of the 
covariance analysis estimate for cost growth, since that estimate (and its correlation 
with the neural net output) are accounted for directly in the final neural net estimate 
for cost growth. 

2. The standard deviation for programmatic cost growth is arrived at by including both 
the neural net error and the covariance estimate for standard deviation. This is so, 
because the neural net does not predict standard deviation directly, and, therefore, our 
probability calculations must consider both sources of variation as well as their 
correlation. 
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3. The MSRR-1 expected value and 95% confidence estimates for cost growth are 
significantly altered by this revision to the COMPRE methodology 
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APPENDIX: Neural Net and Regression Models 

A large number of neural net and regression models have been developed in this effort 
The regression models take the form: 

CC = £ k,C a 't P ‘ a s - X r ‘ x 

7=1 


where, 

C = program cost (1996 $M), 

CG = program cost growth (%), 

N = number of terms in regression equation, 
t = program duration (years), 

X - program chromatic number, 

X = COMPRE technology return parameter, 
a = COMPRE technology risk parameter, 
and all other symbols are estimated parameters. 

Note that for n = 1, the resulting form is intrinsically linear through logarithmic 
transformation. All other values for n result in an intrinsically nonlinear regression form 
One intrinsically linear form, which was found to be statistically significant is: 

CG = 7 1 67 V 767 r 3 242 -16.8 

This regression model is intuitively plausible, in that one would expect cost growth to tend 
to increase with program duration and technology risk, and possibly decrease with 
technology return. While the significance level for this model is 0 .01, the correlation 
coefficient is R = 0.57, showing that only about a third of the total statistical variation in 
the data has been explained by this model. Furthermore, the average error (in cost 
growth) for this model is 34.2% per program, which is unacceptably high for this author. 
Figure A-l shows the fit of predicted to actual cost growth for the 31 programs in the 
database 
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A large number of intrinsically nonlinear models (n>1) were also developed, but none 
achieved average errors less than 23% per program 


A number of neural net models were trained to the database of interest. Table A-l shows 
the connection weights for the best model. This model has a correlation of 0.97, and an 
average error of 5% per program. It also fits more lower cost growth (less than 20%) 
programs better than previously considered neural net models. Finally, this model is 
superior to previous neural net models which involved transformations (sometimes called 
squashing) to the covariance outputs prior to neural net input. 

Although several programs are not well-fit (see Figure A-2), this is consistent with the 
presumption that some programs have higher (or lower) achieved cost growth than their 
respective cost, technology, and schedule parameters would indicate, even given perfect 
information. 

Finally, unlike previously considered neural nets, this neural net is also fully consistent 
with the technology risk and return parameters resulting from the COMPRE covariance 
analysis process. The neural net approach outperforms even intrinsically nonlinear 
regression models (which also require a high degree of parameter training), because neural 
nets synthesize information like a simultaneous system of equations, whereas the 
regression approach involves estimating parameters for a single equation, which may not 
be sufficient to capture complex phenomena 



Actual | 
| Predicted 


Figure A-l. Cost Growth Fit for Intrinsically Linear Model 
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Cost Growth (%) 


Table A-l. COMPRE Neural Net Connection Weights 


Hidden 
Layer or 
Output 
Node 


1 

2 

3 

4 

5 

Co 

Groi 

(OUtl 


Cost 

Variable 


Duration 

Variable 



COMPRE 

Risk 

Variable 

(sigma) 

40.82 

19.26 

4.83 

4.67 

-0.85 

N/A 


COMPRE 

Chromatic 

Bias Node 

Return 

Number 


Variable 

Variable 


(lambda) 

(chi) 


43.96 

1.71 

-12.40 

29.63 

-26.92 

6.74 

-9.30 

6.61 

-12.61 

-19.73 

-9.09 

-2.57 

5.67 

-46.66 

2.90 

N/A 

N/A 

-19.99 


Cost 

Growth 

(output 

node) 

17.4 0 

6.58 

0.64 

10.64 

-5.9 6 

N/A 
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